`iree` submodule #733

makslevental · 2024-08-31T18:40:11Z

This PR checks in iree as a submodule instead of using sync_deps.py. I don't know what the original theory of sync_deps.py but this simplifies dev and deploy.

jtuyls

LGTM, but I also don't know why we went for the script in the first place. Maybe @Abhishek-Varma knows?

Abhishek-Varma · 2024-09-02T07:11:07Z

LGTM, but I also don't know why we went for the script in the first place. Maybe @Abhishek-Varma knows?

I don't know but can only speculate based on the doc comment :-

 Casual developers and CI bots invoke this to do the most efficient checkout of dependencies.

Perhaps @stellaraccident can chime in here.

makslevental · 2024-09-02T15:08:57Z

Casual developers and CI bots invoke this to do the most efficient checkout of dependencies.

Sure but I think no one was/is taking advantage of that. Eg I added skipping torch-mlir but only last week and for that, a simpler way is for iree to make torch-mlir opt-in iree-org/iree#18409 .

newling

I don't know what this reason was for doing it with sync_deps.py either, but I can't see why not to do it this way with submodules if it works and has as much flexibility in terms of clone performance. Nice one!

makslevental · 2024-09-02T19:02:54Z

@newling @jtuyls @Abhishek-Varma can I get a re-review. I changed the clone commands/instructions in order to save people time/space (basically there's a way to do what sync_deps was meant to do using just git).

jtuyls

I was looking at the log and I think the --depth 1 flag is not working:

Run git \
  git \
    -c submodule."third_party/torch-mlir".update=none \
    -c submodule."third_party/stablehlo".update=none \
    -c submodule."src/runtime_src/core/common/aiebu".update=none \
    clone \
    --recursive \
    --shallow-submodules \
    --depth 1 \
    -b $BRANCH_NAME $REPO_ADDRESS .
  shell: /usr/bin/bash -e {0}
  env:
    CACHE_DIR: /_work/iree-amd-aie/iree-amd-aie/.container-cache
    CACHE_KEY: linux-build-test-cpp-asserts-manylinux-v[2](https://github.com/nod-ai/iree-amd-aie/actions/runs/10671927949/job/29578539280?pr=733#step:4:2)-733/merge-405
    BRANCH_NAME: makslevental/iree-submodule
    REPO_ADDRESS: https://github.com/nod-ai/iree-amd-aie
Cloning into '.'...
Submodule 'third_party/XRT' (https://github.com/nod-ai/XRT.git) registered for path 'third_party/XRT'
Submodule 'third_party/aie-rt' (https://github.com/Xilinx/aie-rt.git) registered for path 'third_party/aie-rt'
Submodule 'third_party/bootgen' (https://github.com/Xilinx/bootgen.git) registered for path 'third_party/bootgen'
Submodule 'third_party/iree' (https://github.com/iree-org/iree.git) registered for path 'third_party/iree'
Submodule 'third_party/mlir-air' (https://github.com/nod-ai/mlir-air.git) registered for path 'third_party/mlir-air'
Submodule 'third_party/openssl' (https://github.com/viaduck/openssl-cmake.git) registered for path 'third_party/openssl'
Cloning into '/_work/iree-amd-aie/iree-amd-aie/third_party/XRT'...
Cloning into '/_work/iree-amd-aie/iree-amd-aie/third_party/aie-rt'...
Cloning into '/_work/iree-amd-aie/iree-amd-aie/third_party/bootgen'...
Cloning into '/_work/iree-amd-aie/iree-amd-aie/third_party/iree'...
Cloning into '/_work/iree-amd-aie/iree-amd-aie/third_party/mlir-air'...
warning: --depth is ignored in local clones; use file:// instead.
done.

(https://github.com/nod-ai/iree-amd-aie/actions/runs/10671927949/job/29578539280?pr=733)

newling

Looks good, thanks for updating the README.

README.md

makslevental · 2024-09-02T19:31:09Z

I was looking at the log and I think the --depth 1 flag is not working:

Run git \
  git \
    -c submodule."third_party/torch-mlir".update=none \
    -c submodule."third_party/stablehlo".update=none \
    -c submodule."src/runtime_src/core/common/aiebu".update=none \
    clone \
    --recursive \
    --shallow-submodules \
    --depth 1 \
    -b $BRANCH_NAME $REPO_ADDRESS .
  shell: /usr/bin/bash -e {0}
  env:
    CACHE_DIR: /_work/iree-amd-aie/iree-amd-aie/.container-cache
    CACHE_KEY: linux-build-test-cpp-asserts-manylinux-v[2](https://github.com/nod-ai/iree-amd-aie/actions/runs/10671927949/job/29578539280?pr=733#step:4:2)-733/merge-405
    BRANCH_NAME: makslevental/iree-submodule
    REPO_ADDRESS: https://github.com/nod-ai/iree-amd-aie
Cloning into '.'...
Submodule 'third_party/XRT' (https://github.com/nod-ai/XRT.git) registered for path 'third_party/XRT'
Submodule 'third_party/aie-rt' (https://github.com/Xilinx/aie-rt.git) registered for path 'third_party/aie-rt'
Submodule 'third_party/bootgen' (https://github.com/Xilinx/bootgen.git) registered for path 'third_party/bootgen'
Submodule 'third_party/iree' (https://github.com/iree-org/iree.git) registered for path 'third_party/iree'
Submodule 'third_party/mlir-air' (https://github.com/nod-ai/mlir-air.git) registered for path 'third_party/mlir-air'
Submodule 'third_party/openssl' (https://github.com/viaduck/openssl-cmake.git) registered for path 'third_party/openssl'
Cloning into '/_work/iree-amd-aie/iree-amd-aie/third_party/XRT'...
Cloning into '/_work/iree-amd-aie/iree-amd-aie/third_party/aie-rt'...
Cloning into '/_work/iree-amd-aie/iree-amd-aie/third_party/bootgen'...
Cloning into '/_work/iree-amd-aie/iree-amd-aie/third_party/iree'...
Cloning into '/_work/iree-amd-aie/iree-amd-aie/third_party/mlir-air'...
warning: --depth is ignored in local clones; use file:// instead.
done.

(https://github.com/nod-ai/iree-amd-aie/actions/runs/10671927949/job/29578539280?pr=733)

Ya I saw that and that's why I added -shallow-submodules. I'm not 100% on whether this works correctly but 🤷‍♂️

jtuyls

LGTM

README.md

jtuyls · 2024-09-02T19:42:10Z

I was looking at the log and I think the --depth 1 flag is not working:

Run git \
  git \
    -c submodule."third_party/torch-mlir".update=none \
    -c submodule."third_party/stablehlo".update=none \
    -c submodule."src/runtime_src/core/common/aiebu".update=none \
    clone \
    --recursive \
    --shallow-submodules \
    --depth 1 \
    -b $BRANCH_NAME $REPO_ADDRESS .
  shell: /usr/bin/bash -e {0}
  env:
    CACHE_DIR: /_work/iree-amd-aie/iree-amd-aie/.container-cache
    CACHE_KEY: linux-build-test-cpp-asserts-manylinux-v[2](https://github.com/nod-ai/iree-amd-aie/actions/runs/10671927949/job/29578539280?pr=733#step:4:2)-733/merge-405
    BRANCH_NAME: makslevental/iree-submodule
    REPO_ADDRESS: https://github.com/nod-ai/iree-amd-aie
Cloning into '.'...
Submodule 'third_party/XRT' (https://github.com/nod-ai/XRT.git) registered for path 'third_party/XRT'
Submodule 'third_party/aie-rt' (https://github.com/Xilinx/aie-rt.git) registered for path 'third_party/aie-rt'
Submodule 'third_party/bootgen' (https://github.com/Xilinx/bootgen.git) registered for path 'third_party/bootgen'
Submodule 'third_party/iree' (https://github.com/iree-org/iree.git) registered for path 'third_party/iree'
Submodule 'third_party/mlir-air' (https://github.com/nod-ai/mlir-air.git) registered for path 'third_party/mlir-air'
Submodule 'third_party/openssl' (https://github.com/viaduck/openssl-cmake.git) registered for path 'third_party/openssl'
Cloning into '/_work/iree-amd-aie/iree-amd-aie/third_party/XRT'...
Cloning into '/_work/iree-amd-aie/iree-amd-aie/third_party/aie-rt'...
Cloning into '/_work/iree-amd-aie/iree-amd-aie/third_party/bootgen'...
Cloning into '/_work/iree-amd-aie/iree-amd-aie/third_party/iree'...
Cloning into '/_work/iree-amd-aie/iree-amd-aie/third_party/mlir-air'...
warning: --depth is ignored in local clones; use file:// instead.
done.

(https://github.com/nod-ai/iree-amd-aie/actions/runs/10671927949/job/29578539280?pr=733)

Ya I saw that and that's why I added -shallow-submodules. I'm not 100% on whether this works correctly but 🤷‍♂️

Yeah, not sure, it says to add file://..., maybe something to try at some point, but it looks like the clone goes reasonably fast.. at least faster than sync_deps.py!

makslevental · 2024-09-02T20:42:24Z

Yeah, not sure, it says to add file://..., maybe something to try at some point, but it looks like the clone goes reasonably fast.. at least faster than sync_deps.py!

Ya the thing is I have no clue where to add this... Anyway I'll put a TODO there and make an issue

Co-authored-by: James Newling <[email protected]>

Co-authored-by: Jorn Tuyls <[email protected]>

makslevental changed the title ~~[WIP] iree submodule~~ iree submodule Sep 1, 2024

makslevental requested review from jtuyls, newling, MaheshRavishankar, yzhang93 and Abhishek-Varma September 2, 2024 00:52

jtuyls reviewed Sep 2, 2024

View reviewed changes

newling approved these changes Sep 2, 2024

View reviewed changes

makslevental force-pushed the makslevental/iree-submodule branch from abb4ef6 to 7adb0ec Compare September 2, 2024 16:24

makslevental enabled auto-merge (squash) September 2, 2024 16:24

makslevental disabled auto-merge September 2, 2024 16:25

makslevental force-pushed the makslevental/iree-submodule branch 7 times, most recently from 0d5a546 to ad11611 Compare September 2, 2024 18:58

makslevental requested review from jtuyls and newling September 2, 2024 19:02

jtuyls reviewed Sep 2, 2024

View reviewed changes

newling approved these changes Sep 2, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

jtuyls approved these changes Sep 2, 2024

View reviewed changes

README.md Show resolved Hide resolved

[WIP] iree submodule

74d494b

makslevental and others added 9 commits September 2, 2024 14:38

Update README.md

cc595af

Try ignore sub-sub modules

118e537

Update .gitmodules

a51fe43

Update .gitmodules

2af002f

Update .gitmodules

7822bab

Update README.md

d7112f0

Co-authored-by: James Newling <[email protected]>

Update README.md

57b76ca

Co-authored-by: James Newling <[email protected]>

Update README.md

cad1aae

Co-authored-by: Jorn Tuyls <[email protected]>

Update ci-linux.yml

4d41c79

makslevental force-pushed the makslevental/iree-submodule branch from ff12019 to 4d41c79 Compare September 2, 2024 21:38

makslevental enabled auto-merge (squash) September 2, 2024 21:38

Update README.md

d38cf14

makslevental merged commit 065218b into main Sep 2, 2024
5 checks passed

makslevental deleted the makslevental/iree-submodule branch September 2, 2024 22:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`iree` submodule #733

`iree` submodule #733

makslevental commented Aug 31, 2024 •

edited

Loading

jtuyls left a comment

Abhishek-Varma commented Sep 2, 2024

makslevental commented Sep 2, 2024 •

edited

Loading

newling left a comment

makslevental commented Sep 2, 2024 •

edited

Loading

jtuyls left a comment •

edited

Loading

newling left a comment

makslevental commented Sep 2, 2024

jtuyls left a comment

jtuyls commented Sep 2, 2024

makslevental commented Sep 2, 2024

iree submodule #733

iree submodule #733

Conversation

makslevental commented Aug 31, 2024 • edited Loading

jtuyls left a comment

Choose a reason for hiding this comment

Abhishek-Varma commented Sep 2, 2024

makslevental commented Sep 2, 2024 • edited Loading

newling left a comment

Choose a reason for hiding this comment

makslevental commented Sep 2, 2024 • edited Loading

jtuyls left a comment • edited Loading

Choose a reason for hiding this comment

newling left a comment

Choose a reason for hiding this comment

makslevental commented Sep 2, 2024

jtuyls left a comment

Choose a reason for hiding this comment

jtuyls commented Sep 2, 2024

makslevental commented Sep 2, 2024

`iree` submodule #733

`iree` submodule #733

makslevental commented Aug 31, 2024 •

edited

Loading

makslevental commented Sep 2, 2024 •

edited

Loading

makslevental commented Sep 2, 2024 •

edited

Loading

jtuyls left a comment •

edited

Loading